Driver behavior monitoring

ABSTRACT

Systems and methods provide, implement, and use using a computer-vision based methods of context-sensitive monitoring and characterization of driver behavior. Additional systems and methods are provided for unsupervised learning of action values, monitoring of a driver&#39;s environment, and transmitting visual information from a client to a server.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 15/437,646, filed on Feb. 21, 2017, and entitled“DRIVER BEHAVIOR MONITORING,” which is a continuation of and claimspriority to PCT/US2017/013062, filed Jan. 11, 2017, and entitled “DRIVERBEHAVIOR MONITORING,” which claims priority to U.S. Provisional PatentApplication No. 62/277,470, filed on Jan. 11, 2016, and entitled “DRIVERBEHAVIOR MONITORING,” the disclosures of each of which are herebyexpressly incorporated by reference in their entireties.

BACKGROUND Field

Certain aspects of the present disclosure generally relate tointernet-of-things (IOT) devices and applications, and moreparticularly, to devices, systems and methods for transmitting adescriptor of an event, as may be used, for example, in monitoring abehavior of drivers, in unsupervised learning of action values, and inmonitoring an environment of a vehicle.

Background

Vehicles, such as automobiles, trucks, tractors, motorcycles, bicycles,airplanes, drones, ships, boats, submarines, and others, are typicallyoperated and controlled by human drivers. Through training and withexperience, a human driver may learn how to drive a vehicle safely andefficiently in a range of conditions or contexts. For example, as anautomobile driver gains experience, he may become adept at driving inchallenging conditions such as rain, snow, or darkness.

Drivers may sometimes drive unsafely or inefficiently. Unsafe drivingbehavior may endanger the driver and other drivers and may risk damagingthe vehicle. Unsafe driving behaviors may also lead to fines. Forexample, highway patrol officers may issue a citation for speeding.Unsafe driving behavior may also lead to accidents, which may causephysical harm, and which may, in turn, lead to an increase in insurancerates for operating a vehicle. Inefficient driving, which may includehard accelerations, may increase the costs associated with operating avehicle.

Driving behavior may be monitored. Driver monitoring may be done inreal-time as the driver operates a vehicle, or may be done at a latertime based on recorded data. Driver monitoring at a later time may beuseful, for example, when investigating the cause of an accident. Drivermonitoring in real-time may be useful to guard against unsafe driving,for example, by ensuring that a car cannot exceed a certainpre-determined speed.

While current driver monitoring devices may process inertial sensordata, speedometer data, GPS data, and the like, the use of visual data(such as camera data) in driver monitoring systems has been frustratedby technical problems or challenges related to transmitting and/orviewing large amounts of video. Vehicle monitoring devices present othertechnical hurdles in addition to challenges associated with videotransmission and storage. As such, many of the potential benefits ofvision-based monitoring have not yet been realized. Accordingly, certainaspects of the present disclosure are directed to enabling the use ofvisual data in IOT devices, by providing, for example, systems andmethods for transmitting a descriptor of an event. Furthermore, certainaspects of the present disclosure may be directed to monitoring drivingbehavior.

SUMMARY

Certain aspects of the present disclosure generally relate to providing,implementing, and using a method of transmitting a descriptor of anevent. Enabled with certain aspects of the present disclosure,monitoring systems may log data relating to a visual scene. In someembodiments, an enabled system may classify a driver behavior, such asby characterizing the behavior as “good” or “bad”. In accordance withcertain aspects of the present disclosure, monitoring and/orcharacterization of driver behaviors may be automated. Furthermore,certain aspects of the present disclosure may enable a context-sensitiveclassification of driver behavior. The present disclosure also providessystems and methods for unsupervised learning of action values,monitoring of a driver's environment, and transmitting visual dataand/or descriptors of visual data from a client to a server.

Certain aspects of the present disclosure provide a method oftransmitting a descriptor of an event. The method generally includesreceiving visual data from a first camera at a first device; detectingthe event based at least in part on the visual data and a firstinference engine; determining the descriptor of the event; andtransmitting a first data comprising the descriptor from the firstdevice to a second device.

Certain aspects of the present disclosure provide an apparatusconfigured to transmit a descriptor of an event. The apparatus generallyincludes a first memory unit; a second memory unit; a first at least oneprocessor coupled to the first memory unit; and a second at least oneprocessor coupled to the first memory unit; in which the first at leastone processor is configured to: receive visual data from a first camera;detect the event based at least in part on the visual data and a firstinference engine; determine the descriptor of the event; and transmit afirst data comprising the descriptor from the first device to the secondmemory unit.

Certain aspects of the present disclosure provide an apparatusconfigured to transmit a descriptor of an event. The apparatus generallyincludes means for receiving visual data from a first camera at a firstdevice; means for detecting the event based at least in part on thevisual data and a first inference engine; means for determining thedescriptor of the event; and means for transmitting a first datacomprising the descriptor from the first device to a second device

Certain aspects of the present disclosure provide a non-transitorycomputer-readable medium having program code recorded thereon fortransmitting a descriptor of an event. The program code is executed by aprocessor and generally comprises program code to: receive visual datafrom a first camera; detect an event based at least in part on thevisual data and a first inference engine; determine the descriptor ofthe event; and transmit a first data comprising the descriptor to asecond device.

Certain aspects of the present disclosure provide a method of monitoringa driver behavior. The method generally includes receiving visual datafrom a camera at a first device; wherein the camera is affixed to afirst vehicle; and wherein the first device is connected to the camera;determining a classification of a driving behavior based at least inpart on the visual data and an inference engine; and transmitting theclassification to a second device.

Certain aspects of the present disclosure provide an apparatusconfigured to monitor a driver behavior. The apparatus generallyincludes a first memory unit; a second memory unit; a first at least oneprocessor coupled to the first memory unit; and a second at least oneprocessor coupled to the first memory unit; in which the first at leastone processor is configured to: receive visual data from a camera at afirst device; wherein the camera is affixed to a first vehicle; andwherein the first device is connected to the camera; determine aclassification of a driving behavior based at least in part on thevisual data and an inference engine; and transmit the classification toa second device.

Certain aspects of the present disclosure provide an apparatus formonitoring a driver behavior. The apparatus generally includes means forreceiving visual data from a camera at a first device; wherein thecamera is affixed to a first vehicle; and wherein the first device isconnected to the camera; means for determining a classification of adriving behavior based at least in part on the visual data and aninference engine; and means for transmitting the classification to asecond device.

Certain aspects of the present disclosure provide a non-transitorycomputer-readable medium having program code recorded thereon formonitoring a driver behavior. The program code is executed by aprocessor and generally comprises program code to: receive visual datafrom a camera at a first device; wherein the camera is affixed to afirst vehicle; and wherein the first device is connected to the camera;determine a classification of a driving behavior based at least in parton the visual data and an inference engine; and transmit theclassification to a second device.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example of a device for transmitting a descriptorof an event in accordance with certain aspects of the presentdisclosure.

FIG. 2 illustrates an example of a driver monitoring system inaccordance with certain aspects of the present disclosure.

FIG. 3 illustrates the use of global positioning information todetermine a driving behavior.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. However, it will beapparent to those skilled in the art that these concepts may bepracticed without these specific details. In some instances, well-knownstructures and components are shown in block diagram form in order toavoid obscuring such concepts.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the disclosure is intended to cover any aspect of thedisclosure, whether implemented independently of or combined with anyother aspect of the disclosure. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth. In addition, the scope of the disclosure is intended to coversuch an apparatus or method practiced using other structure,functionality, or structure and functionality in addition to or otherthan the various aspects of the disclosure set forth. It should beunderstood that any aspect of the disclosure disclosed may be embodiedby one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the disclosure.Although some benefits and advantages of the preferred aspects arementioned, the scope of the disclosure is not intended to be limited toparticular benefits, uses or objectives. Rather, aspects of thedisclosure are intended to be broadly applicable to differenttechnologies, system configurations, networks and protocols, some ofwhich are illustrated by way of example in the figures and in thefollowing description of the preferred aspects. The detailed descriptionand drawings are merely illustrative of the disclosure rather thanlimiting, the scope of the disclosure being defined by the appendedclaims and equivalents thereof.

Driver Monitoring Systems

Driver monitoring systems and devices may have multiple applications.Insurance companies may desire aggregated driver behavior data toinfluence premiums. Insurance companies may also seek to reward ‘good’behavior and dis-incentivize ‘bad’ behavior as a way of reducing thenumber of loss events across a population of drivers. Fleet owners,likewise, might desire a system capable of classifying driver behaviorsas a part of a program to incentivize their drivers to drive safely andefficiently. Likewise, taxi aggregators may desire a driver monitoringsystems as part of a program to incentivize taxi driver behavior.Furthermore, taxi or ride-sharing aggregator customers may desire accessto past characterizations of driver behavior so that they may filter andselect drivers based on driver behavior criteria. For example, to ensuresafety, drivers of children or other vulnerable populations might bescreened based on driving behavior exhibited in the past. Parents maydesire to monitor the driving patterns of their kids and may furtherutilize methods of monitoring and characterizing driver behavior toincentivize safe driving behavior.

In addition to human drivers, machine controllers are increasingly beingused to drive vehicles. Self-driving cars, for example, may include amachine controller (which may be referred to as a computerized drivingcontroller) that interprets sensory inputs and issues control signals tothe car so that the car may be driven without a human driver or withminimal human intervention. As with human drivers, machine controllersmay also exhibit unsafe or inefficient driving behaviors. Informationrelating to the driving behavior of a self-driving car may be ofinterest to engineers attempting to improve a self-driving car'scontroller, to law-makers considering policies relating to self-drivingcars, and to other interested parties.

Currently available driver monitoring systems may be based on sensordata available in at an ODB-II port, which may not include vision data.For example, a driver monitoring system may rely on inertial and/orodometer data to monitor a driver. Such a driver monitoring system maybe used to maintain a desired driving behavior and/or limit an undesiredbehavior, but may do so in a way that does not depend on the context ofthe driving behavior. In one example, a new driver may be limited todriving below a certain speed based on odometer measurements. Limitingthe speed of a car for a new driver, however, may have unintendednegative consequences. For example, if the driver is blocking anambulance on a narrow road, the driver may be unable to overcome thespeed settings and may block the ambulance.

Some driver monitoring systems may detect driving events based onnon-visual sensors, but may further include a vision sensor to capturevisual data around the time of a detected event. In one example, adriver monitoring system may process inertial sensor data to detectundesired driving behaviors. An inertial event may be an event with adetectable signature on a trace of accelerometer or gyrometer data, sucha transient spike in an accelerometer trace corresponding to a suddenstop by a vehicle. As commercial-grade inertial sensors may be noisy,however, such a system may falsely detect irrelevant inertial events(which may be referred to as “false alarms”) that have a similaraccelerometer trace but that may not correspond to a driving event ofinterest. For example, running over a pothole or a speed-bump may havean accelerometer reading that is similar to that of a small collision.

To mitigate against false alarms, an inertial sensor-based system mayrecord a video clip upon detecting an inertial event, and then the videoclip may be reviewed by a human operator at a later time. Due to theinvolvement of the human operator, such as system may be expensive andcumbersome. In addition, while the video clip may be useful to correctfalse alarms, an inertial-triggered driver monitoring system may fail tonotice a driving event that does not have a reliably detectable inertialsignature. For example, an inertial-based system with a camera may failto detect a driver running through a red light if the driver neitheraccelerated or decelerated through the red light.

While a designer of an IOT device may desire continuous video recordingand storage to provide greater coverage, practical considerations mayfrustrate the utility of such a system. For example, continuouslyrecorded video may be burdensome to store, expensive and time-consumingto transmit over cellular or other wireless networks, and/or impracticalto review by human operators.

Vision-Based Event Detection

Certain aspects of the present disclosure may enable the use of visualdata in IOT systems and devices, such as driver behavior monitoringsystems. Visual data may improve existing ways or enable new ways ofmonitoring and characterizing driver behavior. In some embodiments,visual data captured at a camera affixed to a vehicle may be used as thebasis for detecting a driving event. For example, a driver monitoringsystem enabled in accordance with certain aspects of the presentdisclosure may detect that a driver has run a red light, even if theevent could not be reliably detected from inertial sensor data and/orGPS data.

Several means for determining an event from visual data arecontemplated. To determine that a driver has run a red light, forexample, a first device may be configured to analyze visual data todetect an object. An object detection may refer to producing boundingboxes and object identifiers that correspond to one or more relevantobjects in a scene. In a driver monitoring system, for example, it maybe desirable to produce bounding boxes surrounding all or most of thevisible cars, as well as visible traffic lights, traffic signs, and thelike. Continuing with the example of running a red light, a first devicemay be configured to detect (locate and identify) a traffic light invisual data across multiple frames, including frames in which only aportion of a traffic light may be visible in the field of view of acamera. The event of running a red-light may then be based on a locationof the detected traffic light and its state (such as green or red).

Several means for detecting an event based on visual data arecontemplated. In some embodiments, bounding boxes for objects may beproduced by a neural network that has been trained to detect andclassify objects that are relevant to driving, such as traffic lights,traffic signs, and vehicles. In some embodiments, vehicles may beassigned to one or more of multiple classes, such as a car class and atruck class. If an image contains two cars and a traffic light, forexample, a trained neural network may be used to analyze the image andproduce a list of three sets of five numbers. Each set of numbers maycorrespond to one of the objects (one set for each of the two cars, anda third set for the traffic light). For each set, four of the fivenumbers may indicate the coordinates of the detected object (forexample, the horizontal and vertical coordinates of a top-left corner ofa bounding box surrounding the object, and a height and a width of thebounding box), and one number indicating the class to which it belonged(for example, the cars may be identified with a “1” and the trafficlight may be identified with a “3”).

In one example, a neural network may have been trained to detect objectsbased on a neural network architecture and/or training framework thathas demonstrated a desired performance on an object detection benchmark,such as the PASCAL VOC 2012 dataset. PASCAL VOC refers to a VisualObject Classes challenge organized by the Pattern Analysis, Statisticalmodelling, and Computational Learning project of the European Union. In2012, the challenge featured a training and validation dataset of 11,530images containing 27,450 annotated objects from 20 classes. In thisdataset, the annotations for objects were provided by human labelers,and included a bounding box and a class identifier for each object. Theannotations may be considered similar to the exemplary neural networkoutput format described above, in which a visual scene containing twocars and one traffic light may be represented with three sets of fivenumbers.

An aspect of the PASCAL VOC 2012 challenge invited participants tosubmit computer vision solutions (which may have been trained on theprovided dataset) that could detect (locate and classify) objects in aseparate test (evaluation) dataset. The performance of a neural networktrained to perform object detection may be assessed in part, forexample, by calculating an overlap between a bounding box output from aneural network (or other computer vision solution) and a bounding boxprovided as an annotation by a human labeler on the same image.Performance assessments could be used to compare different neuralnetwork architectures, for example. In addition, for a learning systemsuch as a neural network, the assessments could be used as a trainingsignal. Progress in automated methods of object detection haveprogressed rapidly in recent years. The availability of benchmarkingdatasets, such as the one featured in PASCAL VOC 2012, may have helpedto drive progress in the field. For years after the PASCAL VOC 2012competition ended, researchers continued to explore new trainingmethods, neural network architectures, and the like, and continued totest their work and report performance on the PASCAL VOC 2012 dataset.For example, a Fast Region-based Convolutional Network (Fast R-CNN)method was introduced in 2015, which reported improvements on a measureof object detection performance on the PASCAL VOC 2012.

Returning to the present example of detecting that a vehicle ran a redlight, a neural network model may be configured and trained to detectobjects in a visual scene according to the Fast R-CNN method describedabove, or other competing methods that would be known to persons havingordinary skill in the art. For example, a neural network having anAlexNet or VGG16 architecture may be pre-trained on to perform objectdetection, using, for example, a PASCAL or ImageNet dataset. The networkmay then be further trained using a custom dataset relevant to drivermonitoring, which may contain images from cameras that were affixed tocars, and which may contain annotated cars, trucks, traffic lights,traffic signs, and the like. In addition, or alternatively, the datasetmay contain images that do not have human annotations, which may be usedfor unsupervised or semi-supervised training. In some embodiments, theneural network may be configured to produce bounding boxes and classidentifiers.

Other output representations are also contemplated. For example, theoutput of a neural network may account for a calibration of a camera andmay, instead of (or in addition to) representing a location of an objectin coordinates of a visual field, represent the location of the objectsas a distance from the camera (which may further include an angle and anindication of size). Likewise, objects may be identified as having adistance relative to a specified point that is not the camera, such asthe front bumper of a car. Alternatively, or in addition, the classidentification data may include a probability of the object belonging toone or more classes, and/or may include a portion of visual dataassociated with the bounding box. A portion of visual data associatedwith a bounding box may be the visual data (such as pixel values)corresponding to the inside of the bounding box, such as in a croppedimage corresponding to each detected object. In some embodiments, thecropped image may be down-sampled, and/or may include a surroundingportion of the bounding box. In some embodiments, the visual data may beincluded in the output of an inference engine in the case that thecertainty of the object classification is low. Alternatively, or inaddition, the output of a neural network may include a feature vectorfrom a specified layer of a neural network, or a portion of a featurevector corresponding to the spatial location of the identified objects.

In some embodiments, an event detection may be based on a sequence ofobject detections from more than one video frame (image). In someembodiments, the object detections across multiple frames may be used tomodulate the confidence associated with an object detection in anyparticular frame. For example, if an object is detected with highconfidence in a first and a third frame, but with a lower confidence inan intervening second frame, the confidence of the object detection inthe second frame may be increased based on the output of the first andthird frames. Likewise, if an object is detected in the second frame,but not in surrounding first or third frames, the associated confidencemight be decreased.

In addition to detecting driving events that may not be otherwisedetectable, visual information may be used to classify a behavior in acontext-sensitive manner. Returning to the example of running ared-light, typically, running a red light may be considered a ‘bad’driving behavior. In some contexts, however, such as when a trafficguard is standing at an intersection and using hand gestures to instructa driver to move through a red light, driving through a red light wouldbe considered ‘good’ driving behavior. Additionally, in some contexts, a‘bad’ driving behavior, such as tailgating, may not be the fault of thedriver. For example, another driver may have pulled into the driver'slane at an unsafe distance ahead of the driver. Visual information mayalso improve the quality of a characterization that may be based onother forms of sensor data. Visual data may be used to determine a safedriving speed based on the observed speed of other drivers, for example,as described below. Several additional examples are provided in thefollowing sections.

The costs associated with monitoring and/or characterizing driverbehavior using methods of the present disclosure may be less than thecosts of alternative methods that may use human inspection of visualdata, or that may rely on certain types of sensors such as RADAR orLIDAR.

Apparatus Configured for Transmitting a Descriptor of an Event

In accordance with certain aspects of the present disclosure, visualdata from a camera sensor may be processed at a connected compute devicewith an inference engine. The inference engine may be a neural networkmodel running on a graphics processor (GPU), digital signal processor(DSP), or other hardware accelerator. The neural network, alone or incombination with data-based heuristics programmed to run on anapplication processor, may be used to infer measurements from the visualdata. For example, the neural network may be trained to detect a trafficlight, where detecting a traffic light may include identifying itslocation in the visual field of the camera as well as its state (red,yellow, or green).

According to certain aspects, monitoring a driver behavior may includecollecting measurements relating to the movements of a vehicle inrelation to lane markings on a road, in relation other vehicles, and/orin relation to other objects such as traffic officers, traffic signals,signs, and the like.

In accordance with certain aspects of the present disclosure, aclassification of a driving behavior may be based on one or moremeasurements determined at a first device connected to a vision sensorthat is affixed to a vehicle. In some embodiments, multiple measurementsmay be produced by a first inference engine running at a specializedhardware accelerator connected to a camera, as described above. Themeasurements may then be transmitted from the first device and receivedat a second device, where a number of measurements may be processed by asecond inference engine. The second inference engine may be a neuralnetwork that is trained to classify a driving behavior based onmeasurements from the first inference engine. As with the firstinference engine, the second inference engine may be a neural networkrunning on a GPU, DSP, and the like, and/or a set of heuristicsprogrammed to run on an application processor. The second inferenceengine may be located on a second device within a vehicle, such as at acentralized processing hub. Alternatively, or in addition, the secondinference engine may be located at a remote location, such as at a cloudserver.

FIG. 1 illustrates an embodiment of the aforementioned devices, systemsand methods for transmitting a descriptor of an event. The device 100may include input sensors (which may include a forward facing camera102, a driver facing camera 104, connections to other cameras that arenot physically mounted to the device, inertial sensors 106, car OBD-IIport sensor data (which may be obtained through a Bluetooth connection108), and the like) and compute capability 110. The compute capabilitymay be a CPU or an integrated System-on-a-chip (SOC), which may includea CPU and other specialized compute cores, such as a graphics processor(GPU), gesture recognition processor, and the like. In some embodiments,a driver behavior monitoring system may include wireless communicationto cloud services, such as with Long Term Evolution (LTE) 116 orBluetooth communication 108 to other devices nearby. For example, thecloud may provide real-time analytics assistance. In an embodimentinvolving cloud services, the cloud may facilitate aggregation andprocessing of data for offline analytics. The device may also include aglobal positioning system (GPS) either as a separate module 112, orintegrated within a System-on-a-chip 110. The device may further includememory storage 114.

A driver monitoring system, in accordance with certain aspects of thepresent disclosure, may assess the driver's behavior in real-time. Forexample, an in-car monitoring system, such as the device 100 illustratedin FIG. 1 that may be mounted to a car, may perform analysis in supportof a driver behavior assessment in real-time. In some embodiments,real-time analysis may refer to a configuration in which visual data isprocessed at a rate that meets or exceeds the rate that it is captured.Alternatively, real-time analysis may refer to a configuration in whichat least a sampling of the visual data is processed on a deviceconnected to the corresponding camera soon after it is recorded. Inthese examples, the driver monitoring system, in comparison with asystem that does not include real-time processing, may avoid storinglarge amounts of sensor data since it may instead store a processed andreduced set of the data. Similarly, or in addition, because the systemmay transmit a reduced set of data (for example, descriptors of objectsthat are detected in the visual data), rather than the visual dataitself, the system may incur fewer costs associated with wirelesslytransmitting data to a remote server.

A system enabled in accordance with certain aspects of the presentdisclosure may also encounter fewer wireless coverage issues. Forexample, a device mounted to a truck may have a memory capacity to storethree hours of high-definition video captured by inward and outwardfacing cameras. If the truck is driven across a portion of the countrythat lacks reliable cellular network coverage, it may be unable totransmit a video for more than three hours, or transmission of a videomay be slower than the rate of collection in such an area. In thesecases, the memory may be used to store video until wireless coverageimproves. Still, some video may be lost once the memory capacity isexhausted. In accordance with certain aspects of the present disclosure,a driver monitoring system may be configured to upload descriptors aboutthe contents of the video data instead of the video data. Such a systemmay still be configured to upload video data that is determined to besalient for some purpose. In this example, the memory capacitycorresponding to three hours of high-definition video data may insteadbe used to store multiple days of descriptor data alone with a reducedset of salient video data. Such a system may be less likely to beaffected by wireless coverage issues because the memory storage may besufficient to store data until the truck is again in a location withsatisfactory cellular coverage, or until the truck encounters a Wi-Fiaccess point.

A driver monitoring system, in accordance with certain aspects of thepresent disclosure, may assess a driver's behavior in multiple contexts.In addition, a driver behavior may be scored using more than one metric.FIG. 2 illustrates a system of driver monitoring in accordance withcertain aspects of the present disclosure. The system may includesensors 210, profiles 230, sensory recognition and monitoring modules240, assessment modules 260, and may produce an overall grade 280.Contemplated driver assessment modules include speed assessment 262,safe following distance 264, obeying traffic signs and lights 266, safelane changes and lane position 268, hard accelerations including turns270, responding to traffic officers, responding to road conditions 272,and responding to emergency vehicles. Each of these exemplary featuresis described in detail below. The present disclosure is not so limiting,however. Many other features of driving behavior may be monitored,assessed, and characterized in accordance with the present disclosure.

Speed Assessment

Certain aspects of the aforementioned driver monitoring system may bedirected to assessing whether the driver is maintaining a safe speed262, which may be based on traffic conditions, weather conditions,visibility, and the like. For example, a driver monitoring system, inaccordance with aspects of the present disclosure, may read speed limitsigns 244 using visual input 212 from a camera. The system may determinethe car's speed 246 using one or more outputs of sensors 210 or sensormonitoring modules 240, including GPS 222, car speedometer 220 (usingeither an OBD-II type interface 226 or a visual camera reading of thedash), and/or visual speed assessment based on detected and computedrelative motion of outside objects 256 (such as stationary sign posts).A driver assessment module 260 may then assess whether the driver ismaintaining (or has maintained if the module is run at a later time) asafe speed 262 by computing a function of the two speeds, namely theposted speed limit 244 and the current speed of the driver 246.

In one embodiment, an assessment of driver speed 262 may be reduced to agrade, or score. The grading may include determining (a) if the driveris over the speed limit, (b) if the driver is over the speed limit byvarious amounts, (c) how much the driver is over the speed limit, or (d)a function of the difference of the two speeds. Such a function mayinclude legally defined traffic fine step points.

A driver monitoring system may also include a determination of thedriver's speed relative to speeds of other cars on the road 248. In thisexample, the grade of the driver may be based on the driver's speed 246relative to the speeds of other cars 248, as well as the posted speedlimit (which may be recognized by a traffic lights and signs recognitionmodule 244). The driver may be graded negatively for going too slow ortoo fast. The driver grade may be more favorable if the driver isdriving over the speed limit at a time when a determined percentage ofother cars are also going approximately the same speed.

Furthermore, a driver monitoring system may determine recommended speedsfrom signs 244, such as signs placed near curves or in other cautionareas. In this example, an assessment module may adjust the driver'sgrade as a function of the driver's speed and the recommended speed. Forexample, the driver grade for safe speed behavior may improve if thedriver's speed is below the recommended speed in a given caution area,as well as below the legal speed limit of the road.

In an embodiment of the present disclosure, a driver monitoring systemmay determine based on visible signage 244 and other scene identifiersthat may be recognized 242 that the car is in a safety sensitive area,such as a construction zone, school zone, day-time headlights requiredzone, and the like. In this example, an assessment module 262 may adjustthe grade for safe speed behavior based on the recognition of the safetysensitive area. For example, the driver grade for safe speed behaviormay improve if the driver's speed 246 is substantially below the speedlimit in these zones (based on visible signage 244), even if the flow oftraffic 248 is above the speed limit. In addition, the contribution tothe grade for safe speed behavior may be weighted higher for these moresensitive zones and weighted relatively lower for less safety sensitiveareas.

In another embodiment of the present disclosure, visual 212 or inertialsensors 218 may be used to determine if the vehicle is on an incline,and furthermore, whether it is traveling uphill or downhill. Thisdetermination may be added as an input to an assessment module forgrading driver speed 262. For example, since a vehicle may have moreproblems stopping when going down a steep decline, the grading mayreflect that traveling above the speed limit in these situations mayhave a more detrimental effect on the grade compared to travelling abovethe speed limit on a flat highway.

In another example, a driver monitoring system may be configured todetect and classify the type of vehicle being driven. Alternatively, thesystem may estimate the type of vehicle. In either case, the system mayuse the information about the type of vehicle 234 to select the relevantspeed limit signs 244 on roads with different speeds for differentvehicle types. For example, a road may have a posted 55 MPH speed limitfor trucks and a 65 MPH speed limit for cars.

According to certain aspects of the present disclosure, a drivermonitoring system may also use context to infer speed limits. Forexample, speed limits may be inferred on roads without marked speedlimits or when a driver enters a road and speed limit signs have not yetbeen observed. A visual scene identification module 242 may determinethat the road is a residential road and the GPS 222 may further indicatethe locality. Based on these, the legal residential speed limit, such as25 miles per hour (MPH), may be applied. Similarly, if the scenedetection module 242 observes a school area and children present, thesystem may determine based on the output from the scene detection module242 that the area is a school zone with the corresponding speed limit.The grading may be applied based on these estimated or inferred speedlimits and the vehicle's measured speed 246.

A grading assessment for driver speed 262 may also be weighted based onthe method used to determine the speed limit. For example, thecontribution to the grade for safe speed behavior may be weighted lowerwhen there is low certainty from the scene identification module 242.

A grading assessment may also provide more granular feedback to the userby, for example, breaking out grades associated with different detectedscenes (based on the output from a scene recognition module 242). Thatmay help educate drivers as to whether they are adjusting their behavioraccording to different driving situations.

A driver monitoring system may process stored data at a later time, inaccordance with aspects of the present disclosure. For example, thesystem may keep track of the driver's speed 246 over time and then usean observed speed limit sign 244 to apply the speed limit over therecent past. In this example, the system may use visual 212, inertial218, GPS 22, map, or other indications to determine the length of timethat that the car has been on the same street and under the same drivingconditions so that the observed speed limit would apply.

Safe Following Distance

Aspects of the present disclosure are directed to visually measuring thefollowing distance 264, which is the distance to the vehicle directly infront of the driver's car. Several methods of visually measuring thefollowing distance are contemplated. For example, a mono-camera 102 maybe used to identify the type of vehicle being followed, such as a sedan,van, or semi-truck. In this example, the following distance may be basedon feature sizes, such as width, or the relative feature sizes ofmultiple features associated with each type of vehicle. In anotherexample, a machine learning model, such as a deep neural network, may beused to determine the distance based on the input pixels correspondingto the vehicle ahead. While the preceding examples utilize amono-camera, the present disclosure is not so limiting. In anotherexample, other sensors, such as RADAR, Ultrasound (SONAR), or LIDAR, maybe used to determine the distance to the vehicle ahead. In addition,multiple methods may be combined to estimate the distance.

In an embodiment of the present disclosure, a driver monitoring systemmay determine the speed of the driver's vehicle 246 and the speed of thevehicle ahead 248. The system may then assess the driver's safefollowing behavior 264, and determine a safe following grade as afunction of the distance to the car and the speeds of the vehicles. Inaddition, the system may further determine the speed of other traffic248 and may incorporate the speed of other traffic in the assessment ofthe driver's safe following behavior 268.

In another embodiment of the aforementioned driver monitoring system,the determined following distance may be converted from a unit ofdistance, such as from feet or meters, to a unit of time, such asseconds. In this example, the assessment of safe following behavior 264may be based on this inferred measure of following time. The drivergrade for safe following behavior may be computed as a function of thefollowing distance in time, and may also be based on the estimatedstopping time based on the current speed of the car 246. For example,driving with less than 2 seconds following time while travelling 30 MPH,or driving with less than 5 seconds following time while travelling 65MPH may result in a reduction in the driver's safe following grade.

In the present example, the method of determining the following distancemay involve a computer vision model in accordance with certain aspectsof the present disclosure. For example, the determination of followingdistance may involve recognizing the type of vehicle ahead along withthe make and model of the vehicle, determining dimensions of the vehiclebased on the make and model, computing the observed dimensions of thevehicle, and estimating the distance based on the relationship betweenthe observed and known vehicle dimensions.

Obeying Traffic Signs and Lights

Aspects of the present disclosure are directed to determining the extentto which a driver obeys traffic lights and signs 266, such as stopsigns, yield signs, do not enter signs, and the like. A drivermonitoring system may use visual input 212 from a camera to detecttraffic signs and traffic lights 244. The system may assess the extentto which the driver is following the traffic signs, or has followedtraffic signs in the case that the assessment is made at a later time.This assessment may include recognizing the intersection markings at atraffic light 244, and determining whether the driver was in theintersection before the light turned red. The assessment may alsoinclude determining the distance to the intersection when the lightturned yellow and whether the driver appropriately stopped or passedthrough the intersection. Further, this may be based on the driver'sspeed 246, which in turn may be based on a reading of the speedometer220, or accessed via an OBD-II 226.

In another example of an assessment of obeying traffic signs and lights264, the visual 212 system (alone or in combination with other sensors210) may measure how much time has elapsed between the light turninggreen and the vehicle starting forward. This measurement may be used toassess that the driver is being responsive to traffic signals. In thisexample, the visual signals 212, RADAR, LIDAR, and/or other systems maydetect if there are cross traffic vehicles that are crossing through theintersection or rapidly approaching the intersection. If so, theassessment module 266 may positively grade the driver for waiting at agreen light if the waiting could be interpreted as a precaution toensure that the cross traffic vehicles will safely stop for their redlights. Likewise, a pedestrian may be detected based on visual data, andthe driver may be graded positively for waiting at a green light if thewaiting could be interpreted as waiting for the pedestrian to finishcrossing the street in front of the driver.

The driver monitoring system, in addition to monitoring and assessingdriver behavior 260, may additionally monitor the behavior of trafficlights and other signals and the behavior of other drivers on the road.For example, according to aspects of the present disclosure, the drivermonitoring system may measure traffic light times 244, and determine,for example, that the yellow light duration does not meet the legalminimum duration. This may be used for contesting traffic infractions ornotifying the government.

Aspects of the present disclosure may be directed to monitoring driverbehavior in response to yield signs. Upon encountering a yield sign (orat a later time if the system is not run in real-time), the drivermonitoring system may visually 212 recognize the yield sign 244 and thenobserve the distances to on-coming vehicles. Based on thesemeasurements, the system may make an assessment 266 that positivelygrades the driver for yielding when the on-coming vehicles are tooclose. The system may also positively grade the driver for not yieldingwhen the on-coming separation is beyond a measured margin, since suchbehavior indicates alertness and may result in increased fuel efficiency

Additionally, there may be instances in which observed traffic signs 244do not actually apply. For example, a construction stop sign may be leftby the road accidentally. In such cases, a visual sub-system may usescene detection 242, possibly in conjunction with other methods fordetermining context to assess whether the observed sign actuallyapplies. For example, the driver monitoring system may connect to acloud service via LTE 224 to use crowd sourcing to determine that mostdrivers do not obey the observed traffic sign, and therefore infer thatthis traffic sign should not be considered in an assessment of driverbehavior, particularly an assessment of the extent to which the driveris obeying traffic signs 266. Likewise, based on the combined outputs ofmultiple driver monitoring systems, the cloud service may determine thata particular traffic sign is invalid. The cloud service may then notifyappropriate government or other groups to the inappropriate trafficsign, in accordance with aspects of the present disclosure.

Aspects of the present disclosure may be directed to monitoring andassessing driver behavior at railroad crossings. The driver monitoringsystem may detect the railroad crossing (either visually and/or viamapping services), and may then grade the driver based on specificsettings. For example, school buses may have a policy setting ofstopping at all rail road crossings. In this example, the vehicleprofile 234 may indicate that the driver being monitored is driving abus, and the obey traffic signs and lights assessment module 266 mayincorporate this indication in the assessment of driver behavior.

The visual system 212 may also be used to help determine the car'svelocity 246 and either alone or in conjunction with inertial sensors218 and/or GPS 222 may determine whether and where the car came to acomplete stop. For example, GPS error rates may make it difficult toassess whether the car stopped before or on the railroad tracks whenwaiting for a traffic light, while a visual road object recognitionsystem 256 may more accurately make that determination.

FIG. 3 illustrates an advantage of using visual information to determinewhether a driver came to a complete stop at an intersection, inaccordance certain aspects of the present disclosure. In thisillustration, a first car 302 and a second car 304 arrive at a stop sign306 and each car comes to a complete stop. The first car 302 stopsbefore the crosswalk 308. The second car 304 enters the crosswalk 308before stopping. The intersection is located in an urban area, withlarge buildings 310 and 312 on either side. In addition, there is alarge truck 314 positioned behind the second car 304. The largebuildings and the truck may interfere with a GPS system by blockingdirect line of sight readings from GPS satellites. The first car has aGPS receiver unit 320 installed near the front right portion of thewindshield. The second car has a GPS receiver unit 324 installed nearthe rear of the hood of the car. The GPS unit 320 of the first carcomputes an uncertainty of its location as determined by the GPS readingin the car. The uncertainty is circular ring 324 that is centered on alocation that is close to, but not exactly coinciding with the truelocation of the GPS receiver unit. The GPS unit 324 of the second carcomputes an uncertainty of its location as determined by the GPS readingin the car. The uncertainty is an elliptical ring 326 that is centeredon a location that is close to, but not exactly coinciding with the truelocation of the GPS receiver unit. Because the buildings 310 and 312 areon the right and left sides of the car, the GPS unit of the second carcalculates greater uncertainty in the right/left axis compared to theforward/back axis. Based on the measured uncertainties of the GPS units,it may be more likely that the first car infers that it did not stopbefore the crosswalk and that the second car infers that it did stopbefore the crosswalk, even though the opposite is true in thisillustration. In this instance, a visual input 102 that could recognize244 the traffic sign 306 or other road markings, such as the crosswalk,could give a more accurate estimate of the true position of each carrelative to the crosswalk. In turn, an assessment module that assessesthe extent to which a driver obeys traffic signs 266 could make a moreaccurate assessment based on visual input 212 rather than GPS 222 alone.

Safe Lane Changes and Lane Position

Aspects of the present disclosure may be directed to assessing thequality of lane changes and lane position 268. For example, a drivermonitoring system may use either visual 212, RADAR, LIDAR, or othersystems 210 to determine the relative positions of vehicles around thecar. The driver monitoring system may then assess the driver's aptitudein maintaining a safe location, such as not driving next to cars inadjacent lanes, but rather maintaining an offset in position. Duringlane changes, the driver monitoring system may assess the driver'sability based on the relative distances and speeds of the driver's car246 and nearby cars 248 when changing lanes. In addition, thisassessment may be based on whether and when the driver signaled lanechanges, which may be accessed via the OBD-II 226.

A driver monitoring system may also observe road markings, such as solidyellow lines or dashed yellow lines, and note if the driverinappropriately crosses over solid yellow lines, or solid white lines inthe case that the driver crosses into a car pool lane outside of thedesignated entry locations.

A driver monitoring system may also determine the rate of closure ofcars in adjacent lanes and use that rate of closure to modify the driverassessment. For example, if a driver changes into a lane with afast-approaching car, the distance threshold for an assessment of thesafety of a lane change may be greater than it would have been if theapproaching car were going about the same speed as the driver's car.

The driver monitoring system may be configured to estimate the fault ofan unsafe driving behavior or context. For example, a driver's safefollowing distance behavior may be scored negatively if the driver iscloser than a safe following distance to the vehicle ahead. However, thesystem may determine that another driver swerved in front of the driverand thus caused the unsafe following distance. In this case, a safefollowing assessment may be adjusted for the driver since the unsafedriving behavior may have been a fault of the other driver. A deviceconfigured according to certain aspects of the present disclosure mayinfer a trajectory of the other driver by detecting the position of theother driver's vehicle in relation to the driver. For example, thetrajectory may indicate that the other driver had been in an adjacentlane just a few seconds before the detected tailgating event, and thusinfer that the driver had crossed in front of the driver to cause theevent.

In one embodiment, the driver monitoring system may track lane markings250 or estimate lane separations, determine the car's position withrespect to the lane markings, and assess the extent to which the drivermaintains a safe lane position. For example, a driver may be graded up(or positively) for maintaining a center location, and may me gradeddown (or negatively) for getting too close to lane edges or crossingover lane edges outside of the context of a lane change. In addition, anaccelerometer 218 may be used to detect freeway rumble ridges, infercrossing into the shoulder, and based on the detected rumble ridges, thedriver's behavior may be graded down. Still, the visual system 212 maybe used to adjust the grading if it recognizes a construction zone witha scene recognition module 242 or recognizes construction cones with aroad object module 256 directing vehicles into the shoulder, forexample. Other methods, such as crowd sourcing in the cloud acrossdrivers, which may be accessed via LTE 224, may also be used todetermine exceptions. For example, if many drivers are making the sameviolation at about the same location, it may be determined that anexception condition is likely.

No Hard Turns, Hard Accelerations Hard Starts

In one configuration, a driver monitoring system may assess the driver'sbehavior with respect to fuel inefficient maneuvers, such as hard startsor hard accelerations 270. An assessment module 260 may include anassessment of acceleration 270, which may be based on a measurement ofthe rate of acceleration 252 using one or more of visual 212, GPS 222,inertial sensors 218, or car speedometer 220. The accelerationassessment module 270, in accordance with certain aspects of theaforementioned systems and methods of classifying a driving behavior,may have a threshold on the amount of acceleration and may note aviolation. Alternatively, or in addition, the module may note a seriesof gradations of acceleration behavior. The visual system 212 may helpmeasure a more accurate range of accelerations by visually assessing therate of change of the road way or other stationary objects. In oneconfiguration, the visual system 212 may overcome limitations associatedwith other sensors. For example, accelerometers 218 may have a maximummeasured acceleration, such that the accelerometer may output a clippedsignal in the case that the maximum is exceeded. Likewise, a GPS 222measurement uncertainty may give false alarms and mis-detects. Inanother configuration, a combination of devices and sensors 210 may beused to measure acceleration and thereby assess a violation.

Responding to Traffic Officers

In one configuration, the driver monitoring system may assess thedriver's behavior with respect to responding to traffic officers. Aresponsiveness assessment module 272 may use the camera to visuallyrecognize traffic officers and categorize their hand signals 254. Thedriver monitoring system may then assess the driver's performanceresponding to the traffic officers. This may also be used to assess thedriver's performance in other assessment modules by issuing anexception. For example, responding to a traffic officer 272 may be ahigher priority than obeying traffic signs or lights 260.

Road Conditions, Potholes, Objects

Aspects of the present disclosure may be directed to assessing adriver's behavior in response to road conditions 272, including thepresence of potholes and objects on the road. For example, the drivermonitoring system may classify the road conditions using visual sensor212 information alone or in combination with inertial sensors 218, orpotentially from cloud service provided information via LTE 224. Thedriver speed assessment 262, following distance assessment 264, andother assessments 260 may be modified based on the road conditions. Forexample, the road conditions may influence the determination of a safedriving speed 262, which may be used by other assessment modules 260.The safe driving speed may be based on the posted speed limit 244 andthen scaled (either linearly or multiplicatively) by the roadconditions. Alternatively, the driver monitoring system may determine amaximum safe speed for the recognized road conditions and the safedriving speed may be determined as the minimum of the determined maximumsafe speed for the recognized road condition and the posted speed limit.

In one configuration, the driver monitoring system may visually detectand categorize potholes and objects in the road 256 and may assess thedriver's response to those objects. For example, a responsivenessassessment module 272 may positively grade the driver for avoiding apothole. The monitoring system may also modify the assessment based onthe location of other cars. In this example, the responsivenessassessment module 272 may positively grade the driver for running over apothole if the position and velocity of other cars would have madeavoiding the pothole dangerous. The detection of road objects 256 by themonitoring system may also modify other assessments, such as theassessment of maintaining good lane position. Typically, an assessmentof the driver's maintenance of good lane position would be negativelyaffected by a swerving maneuver. However, an exception would be raisedbased on detecting that the cause of the swerving was to avoidobstacles. In addition, the driver monitoring system may assess thesafety of the avoidance maneuver based on the distance to other nearbyvehicles.

Emergency Vehicles

Aspects of the present disclosure may be directed to assessing adriver's behavior in response to emergency vehicles. For example, thedriver monitoring system may identify the presence of emergency vehiclesusing an inference engine 258 that may detect sirens based on audio 214and/or visual 212 input. The driver monitoring system may then assessthe driver's responsiveness 272, by determining, for example, whetherthe driver pulled over to the side of the road.

In one configuration, the driver's situational responsiveness could beinferred based on audio input 214. For example, road noise may be usedto determine traffic conditions and/or the presence of honked horns.Furthermore, multiple microphones with beam-forming may be used to helplocalize the direction to the emergency vehicle, car, or train horn.

General Grading Features

A driver behavior may be monitored and/or graded repeatedly over time.In an embodiment in which a driver behavior is monitored, descriptors ofa visual scene may be determined at a device connected to a camera inthe vehicle. Most or all of the descriptors may be transmitted to asecond device, even for periods in which the driving behavior cannot bedetected. For example, a system configured to monitor whether the drivercomes to a complete stop at a stop sign may transmit a substantiallycontinuous stream of descriptor data to the cloud, or to another devicesuch as a centralized analytics hub in the vehicle. The monitoringsystem directed to stop sign behavior may therefore take into accountthe frequency with which stop signs are encountered, as well as makeother inferences associated with time periods in which no stop signs arepresent. A driver's grade, which may be based on the aggregateddescriptors, may be plotted over time. The driver grades may also beaveraged to determine a score per time period, such as per hour, day,week, month, or year. The averaging may be an equal weighted arithmeticaverage, or it may be some other averaging over time. The grading mayinclude different weightings for the assessments of different types ofdriving behavior features 260, for example, an assessment of mergingsafely 268 may be weighted higher than an assessment of hard starts 270.

The methods of grading particular behaviors may be modified based ondriving conditions. For example, the weightings of different drivingbehavior assessments may vary based on a recognized context such as acombination of road conditions, traffic 248, and other recognizedfeatures of the environment 242. For example, the weighting over time ofsafe speed behavior may be higher for riskier environments, so that thecontribution of the driver's speed behavior 262 to the overall grade 280will be higher at these times. Examples of riskier environments mayinclude construction areas, poor weather conditions (such as snow), orpoor visibility (such as fog). In one configuration, the weatherconditions may be determined based on visual inference 212, from anatmospheric sensor 216, from a cloud service accessed via LTE 224, or acombination thereof. The driver monitoring system may also use inertialsensors 218 as an input to an inference engine 240 to assess if thevehicle is maintaining or losing traction, and may thus infer slipperyroad conditions and that the safe driving speed behavior assessment 262should have a larger impact on the overall grade 280 at these times.While the previous examples describe varying the weights of differentdriver behavior assessments, other context-based adjustments are alsocontemplated. In one configuration, the weather or environmentconditions may vary the target threshold values within an assessmentmodule 260. For example, the threshold for safe following distance 264,safe speed of travel 262, or safe lane change 268 may be modified inresponse to observed weather or traffic conditions 248.

In one configuration, descriptors of events in visual data and/orclassifications of driving behaviors may be computed or stored foraveraging over specified contexts, such as environments, areas, ortimes. For example, the driver's grade 280 may be based on behaviorsthat were monitored while the driver was in school zones, as determinedby a scene recognition module 242.

Alternatively, or in addition to averaging, the driver monitoring systemmay accumulate a number of events and use these events as the basis fordetermining a grade for the driver 280. An event may be, for example, atime when the driver's speed 246 exceeded 5 MPH over the posted speedlimit 244. The detection of an event may be associated with a resettime, such that new events of the same type will not be detected againuntil after the reset time has elapsed.

The driver monitoring system may also capture a clip of violationevents, or violation events above a given rating. These clips may bereviewed with the driver at a later time, and may include video data aswell as other types of data.

According to certain aspects of the present disclosure, data may betransferred to the cloud from the driver monitoring system, which may beconsidered a client. In one embodiment, the amount of data transfer fromthe client to the cloud may be based on wireless bandwidthconsiderations. For example, the client and/or cloud may actively managethe amount of data transfer to remain under specified bandwidth usagelimits. For example, the client may be configured with a monthlybandwidth allocation. In this example, the client may measure theutilized bandwidth throughout a month, and may adjust a threshold for areal-time event communication trigger as a function of the utilizedbandwidth and the monthly bandwidth allocation. For example, thefunction may include a ratio of the utilized bandwidth to the monthlybandwidth allocation. In one embodiment, the aforementioned drivermonitoring system may be configured to report speed violations. In thisexample, if the ratio of utilized bandwidth to the monthly bandwidthallocation is low, then the threshold for reporting speed violations maybe set to a low value, such as 5 MPH. As the ratio goes higher, thethreshold may be adjusted higher. For example, if the ratio of utilizedbandwidth to the monthly bandwidth allocation goes above 25%, thethreshold for reporting speed violations may be increased to 10 MPH overthe speed limit.

In another example, the resolution of a picture or video clip may beadjusted based on wireless bandwidth considerations. For example, theamount of data transferred between the client and the cloud may beadjusted based on the ratio of a remaining bandwidth in a monthlybandwidth allocation to the size of the monthly bandwidth allocation. Inthis example, if the ratio is high, then a high-resolution video cliprecorded around the time of an event may be sent to the could forfurther processing. If the remaining bandwidth drops, the videoresolution, and hence the video size, may be reduced for the currentdata transfer and/or for future data transfers in the same month. Othermeans for reducing the bandwidth of data transfer may be used, such asreducing the frame rate, adjusting the ratio of predicted picture frames(P-frames) to intra-coded picture frames (I-frames), adjusting an amountof lossy compression applied, or adjusting a trade-off between sendingfull image clips and sending cropped image around objects of interest.

In one embodiment of the aforementioned driver monitoring system, theclient device may send processed analytics to the cloud. For example,the client may transmit descriptors of detected objects and/or a drivergrade to a device in the cloud, or the client may send a driver grade ora descriptor of a detected object along with a confidence value. Inthese examples, the cloud may make a determination to request additionaldata, such as video clips upon which the processed analytics were based.A request for additional data from the client device may be triggered bya request from a human. For example, a client device may send a messageto the cloud that a certain event has occurred, and the cloud may notifythe driver's supervisor of the event. In this example, the driver'ssupervisor may request a video clip associated with the event foradditional review. The cloud may also be configured to request specificclips and data if they correspond to milestone markers. For example, thedevice may communicate data to the cloud indicating that the driver hasreached an extra safe driving status, and the cloud may requestadditional data to verify the assessment. Similarly, the device may beconfigured to communicate to the cloud when the driver has exhibited acertain level of unsafe driving, and the cloud may then requestadditional data.

In one configuration, the system may assign separate grading scores fordifferent events. In this configuration, the system may separate thegrading of driver behavior into first, identifying ‘unsafe’ events, andsecond, determining whether the driver is responsible for those events.The system may determine responsibility by computing as estimate ofdriver fault based on a trained machine-learning model. Referring toAppendix A, a driver grading system may assign scores using areinforcement learning approach. The machine learning may be implementedusing a Deep Neural Network, Support Vector Machine (SVM), or somecombination thereof. Inputs to the machine learning system may befeature vectors over N frames, such as the last N frames prior to thedetection of an unsafe event. At the next time step, the inputs to themachine learning algorithm may be time stepped by M frames such that ifat t0 the inputs are feature vectors from frames 0 . . . N, then at t1,the inputs are from M . . . N+M, and so on. In this example, a featurevector could include a car speed, a lane position, the relative positionof other cars, and/or the relative position or distance to pedestrians.The speed and/or other data for the feature vector may be obtained fromthe OBD-II port. The feature vector may also include GPS data.

Driver grading for an event may be represented as a tuple, with thefirst score representing the severity of the risk, and the second scorerepresenting the level to which the driver is at fault for the ‘unsafe’event. Over the course of operation, the driver grading may output anN-tuple/(event) or N-tuple/(a set of events), where different scoresrepresent performance for different types of driving behavior. As anexample, there could be a score for lane driving, a score fortailgating, a score for obeying traffic signs, etc. In this example,each of these scores is a field of the N-tuple, and each score may havea second field indicating an estimate of the driver's responsibility foreach ‘unsafe’ event.

According to certain aspects of the present disclosure, the amount ofdata transfer from the client to the cloud may be based on wirelessbandwidth considerations across multiple driver monitoring systemclients. For example, the cloud may aggregate a total bandwidth for amonth, and may allocate more or less bandwidth to specific clientdevices based on various metrics. In this example, metrics may include adegree of agreement of client scores with cloud scores, a driver status,or a measure of the overall remaining bandwidth. In this example, thecloud server may choose to reallocate bandwidth from drivers with fewerevents to drivers with more events.

According to certain aspects of the present disclosure, the amount ofdata transfer from the client to the cloud may be adjusted differentlyin response to bandwidth considerations based on the type of event beingcommunicated. For example, the thresholds at which different types ofevents are communicated to the cloud may be adjusted differently. In oneexample, the adjustment of the amount of data transfer may be done insuch as way that some event types stop reporting for the remainder of amonth. For example, if the bandwidth usage ratio reaches 80%, speedlimit violations may stop being reported, although hard stop events maycontinue to be reported. There may also be some events that continue tobe reported even after the monthly limit is exceeded; for example, if acollision is detected the video clip around the collision may be senteven though the data transfer would exceed the allocated limit and mayincur additional transmission costs.

In addition, events that are not transmitted in real-time or nearreal-time may be cached and transmitted later over a different link,such as Wi-Fi, or at a different time, such as the beginning of the nextmonth. If events are cached for transmission at a later time, the cachedevent transmissions may be limited to a certain usage of the nextmonth's bandwidth allocation. For example, the first 10% of cachedevents may be transmitted at the start of a new month. Alternatively,the set of cached events may be graded and ranked, and the most severeevents may be transmitted. Similarly, this approach may be used formanaging the memory usage of cached event clips and data if memorystorage on the device 114 nears full capacity.

Many of the features described above may be determined based on visualindications of stationary objects, such as traffic signs 244. Visualindications of stationary objects may also be crowd sourced via thecloud, in accordance with aspects of the present disclosure. Forexample, cars may send their GPS coordinates to the cloud via wirelessconnectivity such as LTE 110 at points where they detect a change in theroad characterization 242, such as a change of speed limit or start of aconstruction zone. The cloud may keep track of different road segments,and may provide this information to other driver monitoring systems. Inaddition, the driver monitoring system may periodically report the car'sposition to the cloud service, and the service may send back itsestimate of the road features that the car should expect to observe.This may increase robustness of road feature detection. For example, itmay decrease the uncertainty associated with detecting and/orrecognizing traffic signs that are obscured by other vehicles. It mayalso increase robustness of the sensory recognition systems innight-time versus day-time lighting. The crowd sourced system may alsoreduce classification errors overall.

Crowd-sourcing may also be applied to real-time (dynamic) features, suchas weather, visibility, and traffic conditions. In this example, thedriving monitoring system may adjust its grading in real-time inresponse to the crowd-sourced recognition of road conditions.

In one configuration, GPS could be used to correlate car locations tosensor data recordings from an automobile. Alternatively, or inaddition, car locations could be inferred by matching visual features inthe camera data stream. In this example, matching may includeidentifying distinctive intersections or roadways such as iconiclandmarks, or recognizing signage including mile markers and highwaymarkings. Still other methods of determining car location arecontemplated. For example, short-term dead reckoning may be based on theoutput of inertial sensors 218.

Computational Efficiencies

In accordance with certain aspects of the present disclosure, theprocessing 110 of camera inputs (for example a forward facing camera 102or a driver facing camera 104) of an exemplary driver monitoring devicemay be computationally burdensome. Several methods to improve theefficiency of camera input processing are contemplated. In oneconfiguration, a focal zone may be defined based on the measured speedof the vehicle 246. For example, a focal zone may be an area of interestof the visual scene in which objects that may interact with the vehiclein the near term are likely to be found based on the speed of thevehicle. A focal zone may also refer to a range of scales inmulti-resolution processing. For example, if the vehicle is traveling atslow speeds, the focal zone may be configured to search for pedestriansthat would be closer than 150 feet. This may be used to reduce thepedestrian search space and therefore improve camera input processingtimes. The search space may be reduced either spatially (to only look inthe lower areas of the camera input), or the search space may be reducedin scale (to only search for pedestrians larger than a determined size).While the previous example is directed to a reduction in search spacebased on vehicle speed, the present disclosure is not so limited. Thesearch space selection may be based on a number of factors including thevehicle speed 246, the vehicle environment 256, the search objects, andothers.

While the previous example is directed to limiting a visual search tojust searching one zone, other means for increasing computationalefficiency are contemplated. In one configuration, there could beseveral focal zones, for example a near focal zone, medium distancefocal zone, and a far focal zone. In another configuration there may bea focal zone in front of the vehicle, a focal zone near the vehicle butto the side, and or a focal zone farther from the vehicle. In addition,different focal zones could be associated with different priorities, andthe different focal zones may be processed with different levels ofcomputational complexity. For example, higher priority focal zones maybe searched at a higher frequency compared with lower priority zones.Alternatively, or in addition, higher priority focal zones may have morecomplex and accurate search models than lower priority zones.Alternatively, or in addition, higher priority zones may be searched ata finer window step size than lower priority zones. According to aspectsof the present disclosure, the lower priority zones may be the portionsof the image and image resolutions that are not identified as highpriority focal zones.

In one configuration, the image may be to divided into large sections,such as 8 sections, and an inference engine may be run on each of the 8sections to determine if an object, such as a car or person, is presentin each of the 8 sections. The subset of sections in which the objectwas detected may be further subdivided and the inference engine run onthe subdivided sections. This process may be iterated to furthersubdivide the image so that a detection algorithm (such as selectivesearch or multi-window multi-scale, etc) that may be of highercomputational complexity, may be run over the subsections that testedpositive.

In the previous example, the image may be divided into overlappingsubsections. In one configuration, the initial inference engine used onthe first 8 sections may be a convolutional neural network. Theconvolutional operations may be run over the entire image to compute thepresence of the object in each of the 8 sections at the top of theiterative processing chain to further save computational resources byreusing computations associated with the overlapping portions of the 8sections. Likewise, the convolutional operations may be run on each ofthe identified subsections.

Combined Learned Grading System

The aforementioned driver monitoring systems may include a generalassessment system 260 that may be based on a set of modules 240. Acombination of modules may determine the car and environment statususing a mixture of cameras 212, inertial sensors 214, GPS 222, clouddata 224, profile data 230, which may include vehicle 234 and driverprofiles 232, and other inputs 210. These inputs may then be the basisof a plurality of inferences 240 and grades 260 based on engineeredrules.

Alternatively, or in addition, the grading system 260 that is based onthe car and environment status determining systems may be a machinelearned model that was trained to match expert human labeled gradingscores to environment values. The machine learned model may be appliedto obtain a subset of grading values 260 or a combined value 280. Incomparison to engineered rules, the machine learned grading system mayevaluate grading features for complex scenarios that may not have beenconsidered by the engineers that designed the behavior assessment rules.

In one configuration, the visual system 212 and grading systems 260 and280 may be trained end-to-end. Rather than training the visual systemfor cars 248, pedestrians, lanes 250, distances, and so forth, and thena system on top to match grading scores 260, the system may be trainedend-to-end such that grading scores 260 are computed directly fromsensory data 210. Still, the training procedure may start with trainingcertain subsystems 240 independently, and then performing fullend-to-end training on a combination of subsystems 240 and sensoryinputs 210, such as by back propagation. This training procedure mayresult in the detection of more fine-tuned visual features which may bethe basis for more accurate driver assessment scores 260.

In some embodiments of certain aspects of the present disclosure, adriving behavior may be classified directly based at least in part onvisual data from the camera. This configuration contrasts with someexemplary embodiments described above that may first detect an event anddetermine a descriptor of the event at a first device, then transmit thedescriptor to a second device where it may be used to classify a drivingbehavior. Instead, in some embodiments, the step of determining adescriptor may be skipped. For example, a camera may be affixed to avehicle, and a device may be attached to the camera through a sharedmemory, a wired connection, a wireless connection such as Bluetooth, andthe like. A first device may receive visual data from the camera and maythen directly classify a driving behavior based on the visual data andan inference engine. The inference engine in this example may be neuralnetwork that was trained in an end-to-end fashion, as described below.The inference engine may compute a classification directly from visualsensor data, and may further include as an input other sensor data. Forexample, an end-to-end inference engine may take raw inertial sensordata as an input, may take calibrated inertial sensor data as an input,or may take processed sensor data, such as location estimates from a GPSmodule. Accordingly, the system or device may determine theclassification of the driving behavior at the same device that receivedvisual data from the camera. The classification data may then betransmitted to a second device, where it may then be used for a varietyof applications, including driver monitoring.

In comparison to a system that may split the classification of a drivingbehavior across more than one inference engine, and system having anend-to-end inference engine may be considered to have some advantagesand some disadvantages.

A split system may achieve a desired classification performance based onless training data in comparison an end-to-end system. For example, asplit system may be include separate modules, each of which may detectevents relating on one aspect of driving behavior. For example, onemodule may detect other cars or trucks on the road, while a secondmodule may detect traffic lights. In some embodiments, these two modulesmay share some common processing steps, such as a common trunk of a deeplearning network. Each of these modules may be trained with a number ofexamples covering a variety of relevant objects that might beencountered. For example, a car and truck detecting inference engine maybe trained on labeled car and truck data. In contrast, an end-to-endsystem may be considered to achieve a desired classification performanceafter it may reliably classify driving behavior as safe or unsafe.Relevant training data (such as sensor data corresponding to unsafedriving scenarios) may be less frequently encountered in comparison to,for example, visual data containing other cars or trucks. For thisreason, an end-to-end inference engine may be more challenging to train.

In comparison to a split system, an end-to-end system may transmit lessdata. This may be desirable, for example, if there are many systems inaccordance with the present disclosure deployed by an operator, such asa fleet manager. The additional data from a split system, however, mayhave additional utility. For example, while it may be used formonitoring driver behaviors, the descriptors of objects transmitted froma deployed split-system may more readily be used for additionalpurposes. For example, the additional data may be used to computetypical patterns of observations and may thereby identify an occurrencea rare event for which a classifier has not yet been trained.

Considering the trade-offs between split and end-to-end inferenceengines, it may be desirable to employ a split inference engine whenrelevant training data is scarce, and then gradually adopt a moreend-to-end inference engine as the availability of training dataincreases.

Unsupervised and Reinforcement Learning

In addition to, or instead of, matching human labeled driver assessmentscores, unsupervised and reinforcement learning may be used to lessen oravoid laborious human labeling. These approaches may help create adriver monitoring system that is robust in cases in which human labeledassessments may be in conflict with each other. Examples of such casesinclude swerving to avoid obstacles in the road, passing a constructionstop sign that is not supposed to be active, or following a trafficofficer's directions over road signage.

In one configuration, hand coded rules could be used to determineinitial training values for initializing a system. The system may thenbe further trained and updated using reinforcement learning.

In one configuration, a crowd sourced reinforcement learning approachmay be used. In this example, the driver monitoring system may crowdsource issued traffic infractions across drivers, and use those to labelunsafe or inappropriate driving actions. The visual system may alsoidentify police cars. A reinforcement learning rule may use trainingexamples in which drivers were near police cars. The learning rule maybe based on rewarding the driver behaviors for which the drivers did notreceive an infraction, and penalizing the driver behaviors for which thedrivers did receive an infraction. Similarly, if a driver gets in anaccident, the sensor data preceding the accident may be used as an inputto a reinforcement learning rule to adjust the grading system inferenceengine.

In one configuration, the infractions, traffic accidents, and the likemay be used as a supervisory signal and crowd sourced to train aninference engine that outputs an estimate of the probability of atraffic accident or infraction based on the input sensor data. Anupdated model may be pushed out periodically to the driver monitorsystems to improve the quality of the systems. In this example, theupdated model may be used to update or replace an inference engine onthe device. Configured with an updated inference engine, an enableddevice may exhibit improve driver behavior monitoring. For example, itmay subsequently upload a higher ratio of relevant to irrelevant videoclips to a cloud server.

According to certain aspects of the present disclosure, the drivermonitor system may also utilize a prediction engine to forecastdifferent paths that the driver could take or could have taken in agiven situation, and compute the infraction and/or accidentprobabilities along those paths, and then score the driver based on thepath taken compared with the other paths. The score may be compared withthe best path, the top percentile best path, the average path, or othermeasures.

In one configuration, the driver assessment score may be the Q-score fora Q-learned reinforcement learning model. For example, if the expectedscore is based on negative points for infractions and accidents, thenthe driver monitoring system may note if the expected future score goesbelow a given threshold indicating a higher expected negative event.

In one configuration, the driver monitoring system may assume that thetypical driver behavior in a given situation is a good behavior. Forexample, an inference system trained to classify environments could berun on the inputs to get a feature vector, and the feature vectors couldbe clustered, such as by k-means clustering. This clustering of featurevectors associated with environments may give an unsupervisedcategorization of the different crowd sourced environments. For example,the set of data for a given cluster may be considered the features of anenvironment based on an unsupervised categorization. Based on thecategorization of the environment, the next frames of sensor data, whichmay include video frames corresponding to the car's position in 5seconds, could be crowd sourced and clustered. These clusters may beconsidered action clusters, as they reflect the driver's actions in agiven environment. Assuming that the typical driver behavior is a goodbehavior, the action clusters could be categorized based on the numberof elements in each cluster, which would reflect the likelihood that adriver in a particular environment took a certain action. The actionclusters with more elements or weight/mass may be deemed safer actions.

According to certain aspects of the present disclosure, crowd sourceddriver statistics may be used to determine safe and typical behaviors.In one embodiment, driver behavior may be assessed based on thesedetermined safe and typical behaviors. For example, the driver behaviordistributions may be determined for different maneuvers and events.Statistical analysis or machine learning methods may be used based onthe determined driver behavior distributions to determine driver grades.For example, the lane position for a driver may be measured acrossdrivers, either in general or based on scene recognition determinedsimilar contexts, such as United States freeways. The distribution ofthe lane positions may then be formed with the mean and standarddeviation computed, and the driver grade may be a function of thestandard deviation and mean or median, such as the number of standarddeviations away from the mean. A value closer to the mean may be gradedhigher while a value farther from the mean would be graded lower.Furthermore, the grade may be scaled by the standard deviation of thedistribution of lane positions across drivers in a similar context.Similarly, the distribution of lane changes or freeway merge temporalpaths may be computed across drivers. Then the grade may be based on adeviation from typical paths. Alternatively, the paths across driversmay have grades assigned based on how close they come to anothervehicle. In these examples, the driver's path may be compared against adictionary of paths aggregated across multiple drivers and a grade maybe assigned based on the typical outcome of drivers with similar paths.

As in the above examples, crowd-sourced data may include descriptors ofevents or objects detected by a number of devices. For example, a deviceconfigured according to certain aspects of the present disclosure maytransmit data relating to a car's lane position to a second device. Thesecond device may be a cloud server. The cloud server may also interactwith other similarly configured devices from other cars and may therebyreceive data regarding lane positions, for example, from a sample ofcars on the road. Based on the received data, the second device maycreate an inference engine. For example, it may train a neural networkto detect abnormal lane positions, where a cluster of received laneposition data may be used to determine what a normal lane position couldbe.

Certain aspects of the present disclosure may be directed to systems,devices, and methods of learning action values using Q-learning.Continuing with the above example, an inference engine determined at asecond device may learn that some abnormal lane positions are actuallycharacteristic of safe or responsive driving. In this example, thesecond device may have trained a reinforcement learning model that mayoutput a safe or unsafe action based on visual data, and an inferredaction of the driver may be compared to the output of the model.Alternatively, or in addition, the reinforcement learning model mayoutput a description of a visual scene corresponding to a safe actionbeing taken in response to a first description of the visual scene. Insome embodiments, the reinforcement learning model may be based on asequence of previously detected objects that were detected in visualdata by a deployed device. A reinforcement learning model may be trainedand/or updated based on a human operator agreeing or disagreeing with anoutput of the model. Likewise, an update may be based on an agreementbetween the output of the reinforcement learning model and correspondingoutputs of an ensemble of similarly trained reinforcement learningmodels.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processorfunctionality and external memory providing at least a portion of themachine-readable media, all linked together with other supportingcircuitry through an external bus architecture. Alternatively, theprocessing system may comprise one or more specialized processors forimplementing the neural networks, for example, as well as for otherprocessing systems described herein.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means (e.g., RAM, ROM, a physical storage mediumsuch as a compact disc (CD) or floppy disk, etc.), such that a userterminal and/or base station can obtain the various methods uponcoupling or providing the storage means to the device. Moreover, anyother suitable technique for providing the methods and techniquesdescribed herein to a device can be utilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes and variations may be made in the arrangement, operation anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method comprising: initially receiving, by atleast one processor of a computing device, first visual data captured byat least one camera associated with a monitoring vehicle; identifying,by the at least one processor, a traffic control device from the firstvisual data, wherein the traffic control device is a traffic light orstop sign; receiving, by the at least one processor, second visual datacaptured by the at least one camera, wherein the second visual data iscaptured by the at least one camera subsequent to the capture of thefirst visual data; identifying, by the at least one processor, thetraffic control device from the second visual data based at least inpart on a spatial location of the traffic control device within thefirst visual data and the second visual data; determining, by the atleast one processor, at least one change to an aspect of the trafficcontrol device based on a comparison of the first visual data and thesecond visual data; determining, by the at least one processor, anoccurrence of a driving event comprising running a red light of thetraffic light or failing to stop at the stop sign, wherein theoccurrence of the driving event is determined at least in part based onthe at least one change to the aspect of the traffic control device;determining, by the at least one processor, that a traffic officer ispresent during the occurrence of the driving event based on at least oneof the first visual data or the second visual data; and determining, bythe at least one processor, that the traffic officer caused theoccurrence of the driving event.
 2. The method of claim 1, furthercomprising: determining, by the at least one processor, that theoccurrence of the driving event has occurred for a plurality of vehiclesin a same location; and storing, by the at least one processor, anexception condition in a memory based on the determination that theoccurrence of the driving event has occurred for the plurality ofvehicles.
 3. The method of claim 2, wherein, based on the exceptioncondition, any subsequent vehicle that runs the red light of the trafficlight or fails to stop at the stop sign after the exception conditionhas been stored is determined not to be at fault for running the redlight or failing to stop at the stop sign.
 4. The method of claim 1,wherein the occurrence of the driving event comprises the running of thered light, and further wherein the method comprises: identifying, by theat least one processor, an intersection from at least one of the firstvisual data or the second visual data; and determining, by the at leastone processor, that the red light of the traffic light has been run bydetermining that the monitored vehicle was not in the intersectionbefore the traffic light turned red.
 5. The method of claim 1, whereinthe occurrence of the driving event comprises the failing to stop at thestop sign, and wherein the determination that the monitoring vehicle didnot stop at the stop sign is made based at least in part on videocaptured by the at least one camera.
 6. The method of claim 1, whereinthe determination that the traffic officer caused the occurrence of thedriving event comprises determining, by the at least one processor, thathand signals made by the traffic officer directed the monitored vehicleto run the red light or to not stop at the stop sign.
 7. The method ofclaim 1, wherein the change to an aspect of the traffic control devicecomprises at least one of: a change in illumination of one or morelights of the traffic light; a change in feature size of the trafficcontrol device, a change in an angle of incidence of the traffic controldevice, a change in distance to the traffic control device, or a changein dimension or shape of a determined bounding box for the trafficcontrol device.
 8. The method of claim 1, wherein the occurrence of thedriving event comprises the running of the red light, and furtherwherein the method comprises: identifying, by the at least oneprocessor, an intersection from at least one of the first visual data orthe second visual data; determining, by the at least one processor, thatthe monitored vehicle passed through the intersection, wherein theoccurrence of the driving event is further determined based on thedetermination that the monitored vehicle passed through theintersection.
 9. The method of claim 1, wherein the first visual dataand the second visual data are captured by a plurality of camerasmounted on or in the monitoring vehicle.
 10. The method of claim 1,wherein the occurrence of the unsafe driving event is further determinedbased on at least one of speed data, accelerometer data, gyrometer data,inertial sensor data, atmospheric sensor data, on-board diagnostic(OBD-II) data, or global positioning system (GPS) data.
 11. The methodof claim 1, wherein the occurrence of the driving event comprises thefailing to stop at the stop sign, and further wherein the failing tostop at the stop sign comprises substantially failing to stop at thestop sign.
 12. The method of claim 1, wherein the occurrence of thedriving event comprises the failing to stop at the stop sign, andfurther wherein the failing to stop at the stop sign comprises failingto make a complete stop at the stop sign.
 13. An apparatus comprising: amemory; at least one processor operatively coupled to the memory; and afirst set of instructions stored on the memory and configured to beexecuted by the at least one processor to cause the at least oneprocessor to: initially receive first visual data captured by at leastone camera associated with a monitoring vehicle; identify a lane markingfrom the first visual data; receive second visual data captured by theat least one camera, wherein the second visual data is captured by theat least one camera subsequent to the capture of the first visual data;identify the lane marking from the second visual data based at least inpart on a first spatial location of the lane marking within the firstvisual data and the second visual data; determine at least one change toan aspect of the lane marking based on a comparison of the first visualdata and the second visual data; determine an occurrence of a drivingevent comprising crossing a lane marker, wherein the occurrence of thedriving event is determined at least in part based on the at least onechange to the aspect of the lane marking; determine that a pothole orobject on a road is present during the occurrence of the driving eventbased on at least one of the first visual data or the second visualdata; determine a second spatial location of the pothole or the objecton the road within at least one of the first visual data or the secondvisual data; and determine that the pothole or the object on the roadcaused the occurrence of the driving event based at least in part on acomparison of the first spatial location of the lane marking with thesecond spatial location of the pothole or the object on the road. 14.The apparatus of claim 13, wherein the object on the road is aconstruction zone traffic control device.
 15. The apparatus of claim 13,wherein the at least one processor is further configured to: determinethat the occurrence of the driving event has occurred for a plurality ofvehicles in a same location; and store an exception condition in amemory based on the determination that the occurrence of the drivingevent has occurred for the plurality of vehicles.
 16. The apparatus ofclaim 15, wherein, based on the exception condition, any subsequentvehicle that crosses the lane marking after the exception condition hasbeen stored is determined not to be at fault for crossing the lanemarking.
 17. The apparatus of claim 13, wherein the change to an aspectof the lane marking comprises at least one of: a change in feature sizeof the lane marking, a change in an angle of incidence of the lanemarking, a change in distance to the lane marking, or a change indimension or shape of a determined bounding box for the lane marking.18. A non-transitory computer readable medium having instructions storedthereon that, upon execution by a computing device, cause the computingdevice to perform operations comprising: initially receiving firstvisual data captured by at least one camera associated with a monitoringvehicle; identifying a monitored vehicle from the first visual data;receiving second visual data captured by the at least one camera,wherein the second visual data is captured by the at least one camerasubsequent to the capture of the first visual data; identifying themonitored vehicle from the second visual data based at least in part ona spatial location of the monitored vehicle within the first visual dataand the second visual data; determining at least one change to an aspectof the monitored vehicle based on a comparison of the first visual dataand the second visual data; determining a relative speed of themonitoring vehicle with respect to the monitored vehicle based at leastin part on the at least one change to the aspect of the monitoredvehicle; determining, based on the relative speed of the monitoringvehicle exceeding a predetermined threshold, an occurrence of an unsafedriving event comprising traveling at an unsafe speed; and determiningwhether the monitoring vehicle or the monitored vehicle caused theoccurrence of the unsafe driving event.
 19. The non-transitory computerreadable medium of claim 18, wherein the unsafe speed comprises therelative speed being above a predetermined threshold such that themonitoring vehicle is driving much faster than the monitored vehicle.20. The non-transitory computer readable medium of claim 18, wherein theunsafe speed comprises the relative speed being above a predeterminedthreshold such that the monitoring vehicle is driving much slower thanthe monitored vehicle.
 21. The non-transitory computer readable mediumof claim 18, wherein the instructions further cause the computing deviceto perform operations comprising: identifying a plurality of monitoredvehicles from the first visual data and the second visual data;determining a relative speed of each of the plurality of monitoredvehicles with respect to the monitoring vehicle; and determining thatthe monitoring vehicle caused the occurrence of the unsafe driving eventwhen the monitoring vehicle is traveling faster than a predeterminedpercentage of the plurality of monitored vehicles.
 22. Thenon-transitory computer readable medium of claim 18, wherein thedetermination whether the monitoring vehicle or the monitored vehiclehas caused the occurrence of the unsafe driving event is based at leastin part on a determination that the monitoring vehicle is in a safetysensitive area.
 23. The non-transitory computer readable medium of claim18, wherein the determination whether the monitoring vehicle or themonitored vehicle has caused the occurrence of the unsafe driving eventis based at least in part on a determined recommended speed in an area.